This invention relates in general to electronic data handling devices and, more particularly, to interactive electronic data storage and processing devices having integrated data displays. Specifically, the invention is directed to a portable, interactive electro-optic data input/output, storage, processing, and display device responsive to hand printed text and hand drawn graphics, as well as hand entered commands, input directly onto the display surface, for providing an electronic writing and drawing slate. The device also provides an input/output capability with other information processing equipment, such as a plotter or computer.
Various display systems are known which incorporate the capability for pointing or writing directly onto a display screen, along with a detector for sensing and encoding the location of the user's finger or a writing stylus on the display screen so that graphic information can be drawn directly onto a display surface. These display systems include low resolution (e.g., touch screen) devices, high power (e.g., CRT, light emitting diode, and electroluminescent) devices, and non-emissive mechanical displays. These display systems have various shortcomings.
Touch screen and other display systems utilizing separate hardware on the display surface for sensing the position of a writing stylus (or finger), such as a dedicated sensing layer, are typified by input resolution lower than desirable for interactive input and processing of graphics. Input registration and parallax present problems for interactive graphics when physically separate sensing and display elements are employed at resolutions desirable for detailed graphics. Limits on miniaturization of sensing hardware and dedicated interconnecting lines for addressing individual sense locations also place restrictions on achievable input resolution. Additionally, fabrication costs and complexity are increased by the need for separate dedicated sensing hardware on the display surface, as well as requirements that the sense locations be interfaced with display hardware for properly associating the sense locations with the appropriate display elements.
CRT's and display systems employing other emissive (e.g., LED and electroluminescent) displays in connection with a light pen do not require auxiliary sensing hardware on the display surface. The need for auxiliary sensing hardware on the display surface is eliminated by providing a light receiving pen which senses light emitted from the display elements. The optical output from emissive display elements detected by the light pen is then used to determine pen position. See U.S. Pat. No. 4,268,826. Since these display systems use emissive display elements, however, they consume more power than desirable for a portable battery-operated device.
Non-emissive mechanical displays incorporating a writing instrument employ mechanical display elements moved through a fluid whenever the state (i.e., "on" or "off") of any of these elements is changed. See U.S. Pat. No. 4,520,357. These mechanical displays require greater operating power than other non-emissive electronic (e.g., liquid crystal) displays. Additionally, the writing instrument changes the state of display elements by moving them physically, while leaving the previously stored state of any touched display element in memory unchanged. Input to memory is effected by detecting the state of each display element after input by the writing instrument, and then comparing the detected state to the state previously stored in memory. The memory can then be updated to reflect any inputted change in state. Since input depends on changing the state of the display elements, and since the writing instrument can change only "on" display elements to "off", or vice versa, the user is able to effect input only to one of either "on" or "off" display elements at any one time. This is a substantial limitation to interactive word processing and graphics, as well as to other input operations, which desirably accommodate input to arbitrary display element locations (regardless of whether they are "on" or "off").
Additionally, low power electro-optic displays suitable for displaying text and graphics are known. These displays can be constructed with high pixel densities including surface areas containing many thousands of individually addressable pixel display elements. The compact nature, flat profile, light weight, and low power requirements of some of these displays have contributed at least some degree of portability in such products as laptop personal computers which embody a flat panel information display with sufficient pixel density to display graphics and with a large enough active surface area to display multiple lines of text. Unfortunately, these displays are typically associated with additional components, such as keyboards, which detract from the overall compactness and portability and do not accommodate direct hand entry of data onto the display surface. Further problems associated with using known techniques in connection with a high resolution, portable data input/output, storage, processing, and display device will become apparent in view of the following observations.
Low power consumption is essential, in the context of portability, to facilitate operation for long periods between battery replacements, as well as the use of inexpensive and readily obtainable batteries which occupy minimal space. Additionally, low power consumption enables the use of other modes of compact and portable power, such as solar cells. Also, low power consumption enables storage of large quantities of information, for longer periods, in low cost, high capacity, volatile memory.
Non-mechanical, non-emmissive displays (e.g., LCD's) offer lower power consumption and lower cost than other known flat panel devices and can be produced by well known manufacturing processes and with high resolution active matrix addressing for high contrast and wide angle viewing. Among the problems in employing known non-mechanical, non-emissive flat panel displays (notably, LCD's), however, is that slight pressure applied by a writing stylus onto the display surface causes minute changes in the thickness of a fluid-filled cavity containing an electro-optically active material, such as a liquid crystal, resulting in display distortion. This problem can be lessened by utilizing a dedicated stylus sense substrate superimposed directly above, and parallel to, the display surface, and which is not in physical contact with the display surface. Such a solution, however, increases fabrication complexity and costs, as well as introduces input registration and parallax problems undesirable in the case of interactive graphics at high resolution.
Another problem in employing non-mechanical, non-emissive displays (e.g., LCD's) relates to detection of writing stylus position. By way of contrast, mechanical and emissive displays with pen input avoid the need for auxiliary sensing hardware on the display by employing display elements themselves as pen sense points. On the one hand, the mechanical displays achieve this by providing an input writing instrument which physically toggles the position of display elements in a way that is both visible as a change in display state (e.g., "on" to "off"), and detectable, by display electrodes as a local capacitance change. In order to effect input, the mechanical displays compare display element states before writing instrument input (stored in memory) and after writing instrument input. Thereafter, any inputted changes to the display are updated in memory. On the other hand, emissive display systems eliminate the need for auxiliary sensing hardware by utilizing a light pen with the ability to sense the optical output of emissive display elements themselves. In contrast, a non-mechanical, non-emissive (e.g., LCD) display has neither mechanical display element members which can be toggled, nor light emitting display elements which can be efficiently and reliably sensed by a light pen.
An additional problem in a non-mechanical, non-emissive display relates to the speed with which writing stylus input must be sensed in order to accommodate high input resolution without auxiliary sensing hardware on the display. Mechanical displays and emissive display systems, which use the display elements themselves as pen sense points, perform the sensing function by sampling each pen sense location on a display surface, either in concert with display output, or in a separate cycle, and for each pen sense (display element) location, the presence (or absence) of the writing instrument or light pen is determined. To accommodate a human observer, an interactive display must check all the pen sense locations on a display surface for new input, and then update the display for viewing any new input, within a "frame time" period of roughly 30 milliseconds.
In this regard, the sampling of display elements in mechanical displays and emissive display systems employing display elements as pen sense points is adequate provided there is low enough input resolution that natural pen movement during a frame time is unlikely to traverse more display elements than the number of times each display element location can be sampled for input during a frame time. This is necessary in order to provide sufficiently rapid tracking of arbitrary writing stylus movement. Some known display systems incorporating high power emissive display elements achieve more rapid pen sensing, but only do so at the expense of requiring that added pen sense instrumentality auxiliary to the display itself be fabricated into the display system (e.g., Japanese Pat. No. 58-17247(A), which incorporates a piezoelectric pen sense layer).
In the case of interactive graphics where high pixel resolution (and concomitantly high pen sense element density) is desirable, and given a reasonable pen sense element spacing of, for example, 10 mils, it is conceivable for known displays employing display elements as input points that a writing instrument or light pen could travel from one pen sense element to another in less time than would be possible to sample all pen sense elements on a display. To address this problem, known interactive displays typically incorporate dedicated sensing hardware (e.g., a dedicated sensing layer or an external digitizing system), suffer reduced resolution and/or display area (in order to reduce the number of points sampled in a frame time), or restrict the speed of pen movement.
It is also pointed out that known display systems have heretofore been limited in the amount of information which can be stored internally on electronic memory devices. The limitation is attributable not only to the storage capacity, physical size, power requirements, and cost of available electronic memory, but also by the extent to which the displays can store commonly used symbols, such as text characters and graphic shapes, directly and in a compact and referenceable format, such as ASCII, suitable for word processing and for interface to conventional text processing and printing devices.
Conventionally, typefont text characters, each displayed as a pattern of many individual pixel display elements, can be input to a display via a keyboard device, in a standardized coded format, such as ASCII. This facilitates efficient text storage, word processing, and interface to conventional text processing devices, such as printers, which accept standardized text formats. However, in order for text characters to be thus input in a compact and referenceable way, the additional instrumentality for text character selection, such as a keyboard, is typically employed. The keyboard must be interfaced to the display and display storage means, which detracts from the overall portability of the display system by requiring a separate structure to accommodate the customary array of fingertip sized keypads required for comprehensive keyboard text input, or by occupying the limited surface area on the display itself to accommodate the required text selection instrumentality, thereby increasing size and weight and/or limiting display area needed for enhanced viewing and editing of full page documents.
If, however, text is input directly onto the display surface via a writing stylus, then hand written textual information can be input, displayed, and stored without requiring text selection instrumentality auxiliary to the display itself. However, substantially more electronic memory is typically required for storing each hand written text character, since each character consists of a multiplicity of pixel display elements, each of which may need to be separately represented in memory, in contrast to text characters represented in a standardized format usually requiring only a single byte for each character stored. For a display input and storage device possessing a notepad or notebook sized display page and which stores hand drawn text or graphics, considerable electronic memory is required for storing more than just a few display pages of information. Even with data compression, such as disclosed in T. S. Huang, et al., "Runlength Encoding and Its Extensions," Symposium on Picture Bandwidth Compression, Massachusetts Institute of Technology, 1969, Gordon and Breach (New York), 1972, the monetary cost of electronic memory severely limits the price/performance, i.e., cost versus storage capacity relationship. Additionally, word processing functions, straightforward with characters represented in a compact and referenceable format, such as ASCII, are much more cumbersome with hand drawn characters. Also, conventional text printing devices are not well adapted to reproduce hand drawn text.
Alternatively, computer terminal devices incorporating a dedicated digitization indicator board equipped with a writing instrument, and utilizing character recognition techniques for input of hand printed commands and text are known. See, e.g., "Micropad Handwriting Terminal," Computer Equipment Review, Vol. 3, No. 1, pages 51-55, and U.S. Pat. No. 4,562,304. These on-line hand print character recognition systems have been found useful in applications where limited amounts of text are routinely input to a computer. For example, these systems are used in lieu of conventional inventory forms or hotel guest sign-in forms.
However, the character recognition techniques associated with these on-line hand print character recognition systems require that adjacent characters be explicitly separated. Accordingly, each character must be drawn within the confines of a prescribed area, or handprint box, printed in grid paper form pre-aligned to a writing panel prior to use. Grid or box entry precludes adjacent characters from overlapping or touching, as is common with natural hand printing.
Another device, the Casio PF-8000, permitting a single character to be printed individually within the confines of an associated digitization panel by means of a user's finger tip, is also known. However, in addition to the above mentioned constraints, this device also prescribes stroke direction (i.e., up or down), as well as the order in which strokes may be drawn when inputting text characters.
In any event, known character recognition techniques provided for on-line hand print character input systems require that characters be printed by prescribed rules which encumber natural hand printing, and consequently make input of lengthy text strings inefficient by comparison to the alternative of keyboard input. Additionally, variations characteristic of natural hand printing, such as stroke overwriting, character skew, serifs, or arbitrarily connected or non-connected strokes, are restricted or disallowed by currently available on-line hand print character recognition devices. Furthermore, the display surface on these devices, being incapable of independently accepting input from a writing instrument, only accepts data from an associated and separate digitization panel which is itself incapable of independently displaying information, so that the writing instrument cannot be used for directing display processing changes or input to a particular character or location within a body of converted text, displayed on the display surface. Word processing instead requires explicit cursor positioning controls responsive to keyboard commands, mousing, or the digitization panel, the added instrumentality of which detracts from portability and complicates operation.
Less restrictive character recognition techniques than those mentioned above are known (e.g., C. C. Tappert, "Adaptive On-Line Handwriting Recognition," Seventh International Conference on Pattern Recognition (Cat. No. 84CH2046-1), Montreal, Que., Canada, July 30-Aug. 2, 1984 (Silver Spring, MD:IEEE Comput. Soc. Press 1984, Vol. 2, p. 1004-1007). However, they are computationally more complex and consequently less cost effective and/or slower than are used in current on-line hand print character input systems.
For a highly portable and interactive data input/output, storage, processing, and display device, a text and high resolution graphics capability is needed that can be operated without requiring auxiliary text selection hardware, such as a keyboard, and without consuming available display surface area for keyboard text selection instrumentality auxiliary to the display itself. Additionally, there should be provided an economical and rapid on-line hand print character recognition capability that allows pages of text to be efficiently input, and stored with a minimum of on-board electronic memory devices, and in a form amenable to word processing and data I/O in association with conventional text processing/printing devices.